Vaccines: Many Beliefs Study 3 - Simulations

as of May 2, 2018 all text in this notebook is just copy/pasted from the study2 notebook, so should largely be disregarded. will fill this in later

A few initial thoughts

Fundamentally, there are two major questions we want to ask/answer:

1. How well can we predict the effect of the intervention on other beliefs using a previously-learned network (or by averaging over networks)?
2. Given what we know about the intervention effect, what is the best model of the belief system?

The first is an out-of-sample prediction question, the second is more a model fit question. The first can be answered by generating predictions from hydenet, or possibly even by generating predictions with BRMS and using rejection sampling (since just conditioning on the assigned condition variable). The second can be addressed with either a cross validation and prediction scheme, or with WAIC via BRMS, either using imputed data or using a two-stage fitting process where posteriors from stage 1 are passed in as priors in stage 2.

In this notebook, I’m going to focus on the first issue: predicting effects of interventions from learned networks.

Predicting scale averages

Let’s see what the model predicts based on the change in the scale averages from pre-test to post-test. To transform this into something more like a real “fit”, one approach is to predict a distribution of change scores and then compare the predicted and observed distributions.

Disease Risk Condition

I’ll generate predictions for each of the 13 other nodes with diseaseSevere = pre-test mean and diseaseSevere = post-test mean, and see how those other nodes change. This is like imagining that all that’s changed between pre- and post-test is the information affecting diseaseSevere beliefs. So, conditioned only on what we know about how that changed, what should we expect would also change?

Once we start visualizing the error inherent in the data and estimates, only the prediction for “diseaseRare”, “overpar”, and “vaxtox” lie more than 1 standard error away from the observed mean. The all-important “vaxIntent” prediction is right about 1 SE away from the observed mean.

That’s part win for the model and part failure of the data to provide enough precision to really assess it strongly.

Below I do the actual inference with mcmc in jags, sampling from the multivariate distribution defined by the model graph.

NAs introduced by coercionattributes are not identical across measure variables;
they will be droppedJoining, by = "ResponseId"
Setting row names on a tibble is deprecated.The working directory was changed to /Users/derekpowell/Projects/Vaccines/Repos/vaccineBeliefs/study1 inside a notebook chunk. The working directory will be reset when the chunk is finished running. Use the knitr root.dir option in the setup chunk to change the working directory for notebook chunks.

Recall, here’s the network …

Predicted change scores against observed change scores (error bars = 95% CI) …

binding factor and character vector, coercing into character vectorbinding character and factor vector, coercing into character vector

bootstrapping CI

Here I’m going to bootstrap a confidence interval for the correlation between model predictions and observations for aggregated change scores across the 13 predicted scales.

I was thinking that it’s not quite right to do your standard “significance” test on this correlation, as these predicted/observed change scores aren’t values sampled from a population of variables. Instead, they represent their own population, but their measurement is uncertain, with that uncertainty coming from the participant-level data.

As I reflect on this I’m potentially a bit out of my depth, statistically. But the idea was I would take bootstrap samples of the data, make model predictions for each, then calculate the correlation with obtained values.

The results give a bit more confidence about the estimate than you’d have from the CI you’d get just treating the 13 points as a sample from a population (.303 to .914).

The mean correlation value is 0.748 and the bootstrapped 95% CI does not include zero. So, minimally, the model is picking up on something. Below is a histogram of correlation values, with a line marking the observed correlation in the original sample.

[1] 0.7477308
     2.5%     97.5% 
0.6259831 0.8553284 

Modeling participant-wise predictions

We can also ask if the difference from pre to post-test for individual participants are similar to what we would expect according to the model. What I’m doing here is generating model predictions based on pretest diseaseSevere and posttest diseaseSevere ratings. Then, computing a predicted change score for each participant on each remaining scale, and adding that to their pretest scores to predict their posttest scores. I’m then truncating within the allowed range of -3 to 3, and recalculating a predicted change score.

So predicting posttest scores from pretest scores + predicted change scores, we do pretty well. The correlations overall seem good, but they are mostly driven by the knoweldge of the pretest scores.

That’s evidenced by the abysmal performance of the model to predict individual participants’ change scores. I am not quite sure how to explain them beyond the fact that the degree of predicted change is clearly dependent upon the prior score. That is, we expect smaller increases with high scores and smaller decreases with low scores, etc. In the case of scales where the model predictions are NEGATIVELY correlated, I think that might be the explanation. Right now the only thing in the model to guard against that is a simply truncation on the posttest predictions, bounding them within the range of -3 to 3. (but that doesn’t seem to affect the prediction/fit very much)

Average correlation between predicted and observed across all scales is approximately r = .1

So the model fails miserably at individual-level predictions, capturing changes only at the population level.

Baseline model (naïve bayes)

To evaluate the performance of our model, we should compare it to something else. I think a good baseline candidate is a “naïve bayes” or naïve causal model–where diseaseSevere predicts all other beliefs. This could be simulated by a series of linear regressions, but I think it will actually be easier to put it into hydenet.

Visually …

binding factor and character vector, coercing into character vectorbinding character and factor vector, coercing into character vector

And performance is essentially identical. That is, if we just naïvely assume disease severity drives all beliefs and we get just about as good a prediction as our more sophisticated model.

Though again, very poor individual-level predictions.

Of course, our learned structure accounts for the original correlational data far (infinitely?) better. As revealed by a \(\Delta AIC\) in the thousands.

[1] "naïve structure AIC = -244822"
[1] "learned structure AIC = -21115.34"

Preliminary conclusions

On some reflection, these results shoudln’t have been so shocking. Since the structure learning algorithm is working to find a graph that preserves the correlations among variables, it’s not so surprising that conditioning on disease severity in the graph produces roughly the same predictions as a set of regression models predicting all other node from disease severity (or equivalently, the “naïve” network I used here). However, the fact that those predictions map reasonably well onto what we observe is still important!

What’s better about the network structure models compared with naïve or univariate models?

1. Fit both correlations between all scales and predict change scores on all scales
2. Predict effects of educational interventions on any node or multiple nodes, all in one model
3. Give ways to help diagnose which nodes were directly affected by interventions
4. (Arguably) better allow interpretation of overall structure of belief system, better provide insights, etc
5. Serve as a cognitive model of the belief system in important ways

If we have three models, one that lets us do prediction in one domain, another that lets us to do prediction in another domain, and one that does both – we should prefer the one that does both. Likewise, if we have three models, one that lets us do prediction, one that provides an intelligible cognitive theory, and one that does both–we should prefer the one that does both. So it would have been (or would be) a lot more exciting if the theory models could outperform the naïve models on both tasks, but it’s still interesting/cool if they do at least as well on multiple tasks as the other models each do on their one task.

That final point is important too. The (presumed) fact that the model will do better prediction treated as a cognitive model than as a causal model is very informative. In a sense, it’s actually supported by the naïve models too. A naïve model where disease severity predicts all beliefs is a very poor model of the overall system. But, it is a decent model of how the beliefs change. This would be very unlikely if what we were modeling was a simple causal system.

This also suggests a more general lesson: If beliefs are sufficiently interconnected and there are few v-structures, then the directions of each edge may matter very little in terms of prediction. Of course, this also means that setting those edge directions (if we care to do so), is going to be difficult. It might also mean it’s not the right way of concieving of the relationships among beliefs.

Going forward

In the next month or two, I think we should try to address a few questions before moving to publish …

1. Can we use more sophisticated techniques to learn a better initial model that outperforms naïve models?
2. Can we use this intervention data to learn a better model that will begin to outperform naïve models?
3. How far can we push “naïve” models in terms of doing predictions?